Accelerated re-encoding of video for video delivery

ABSTRACT

In some embodiments, a method receives an encoded video and decodes blocks in a frame of the encoded video in which replacement content will be inserted. The blocks are analyzed to determine first blocks of the frame that are affected by insertion of the replacement content and second blocks in the frame that are not affected by the insertion. Information from the decoding of the block is saved for the second blocks. The frame is encoded where the replacement content is inserted to replace the first blocks in the encoding of the frame, and wherein the information from the decoding of the second blocks is reused when performing the encoding of the second blocks.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation application and, pursuant to 35U.S.C. § 120, is entitled to and claims the benefit of earlier filedapplication U.S. application Ser. No. 15/836,501 filed Dec. 8, 2017, thecontent of which is incorporated herein by reference in its entirety forall purposes.

BACKGROUND

A video delivery service may insert advertisements (ads) in content thatis delivered to users in different locations. One method of insertingthe ads into the content is to break up the content with ad breaks andinsert video ads in the ad breaks. The ad breaks may be full-screen adsthat include the advertiser content. These full-screen ads may beexpensive to produce, however. Additionally, the ad breaks may disruptthe viewing experience for the user by interrupting the viewing of thevideo content, such as the show or movie being watched.

The users that are viewing the video are located in different locations.Sending the same ad to all the users may not show each user the mostrelevant ad possible. For example, some ads may be more relevant tousers in a first location than to users in a second location. However,the advertisers may not have a choice as to who is offered theadvertisement. Further, by only providing advertisements to all users,some regional advertisers may not want to advertise on the videodelivery service because their ad may not be relevant to large portionsof the video delivery service's users.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a simplified system for delivering video with replacementcontent according to some embodiments.

FIG. 2A depicts an example of a client device at location #1 accordingto some embodiments.

FIG. 2B depicts an example of a client device that is in location #3according to some embodiments.

FIG. 3 depicts a simplified flowchart of a method for replacing contentin videos according to some embodiments.

FIG. 4 depicts a simplified flowchart of a method for insertingreplacement content into a video according to some embodiments.

FIG. 5 depicts of an example a transcoder according to some embodiments.

FIG. 6 depicts a more detailed example of the transcoder according tosome embodiments.

FIG. 7 depicts an example of performing the motion prediction processaccording to some embodiments.

FIG. 8 depicts a video streaming system in communication with multipleclient devices via one or more communication networks according to oneembodiment.

FIG. 9 depicts a diagrammatic view of an apparatus for viewing videocontent and advertisements.

DETAILED DESCRIPTION

Described herein are techniques for a transcoding system. In thefollowing description, for purposes of explanation, numerous examplesand specific details are set forth in order to provide a thoroughunderstanding of some embodiments. Some embodiments as defined by theclaims may include some or all of the features in these examples aloneor in combination with other features described below, and may furtherinclude modifications and equivalents of the features and conceptsdescribed herein.

Some embodiments distribute video to client devices for users located indifferent locations. For example, users may be using client devices indifferent defined areas, such as in different cities, regions, zipcodes, etc. A video delivery system may receive the locations for theclient devices and determine replacement content that is used to replacecontent in the videos. For example, the video delivery system mayanalyze the content of the video to determine replacement areas in thevideos. The replacement areas may be defined by a time period in thevideo and an area in the images of the video. For example, there may beareas in the images during a number of frames of the video, such as abillboard or product in which replacement content can be inserted. Forexample, an advertisement may be inserted and replace the content in abillboard in the video.

The video delivery system may determine different replacement contentfor different locations. For example, advertisements may be targeted todifferent location region, such as an advertiser in the northeasternregion may want to display an ad to users that are only located in thenortheastern region. The video delivery system provides a platform thatallows a content delivery network (CDN) to deliver the location-basedreplacement content in videos. For example, the system may define thecontent replacement areas, which may include a time period and an areain images in the video. Then, the system can specify which replacementcontent should be inserted into the content replacement areas fordifferent locations.

When sending videos to client devices, servers, such as edge servers,receive an encoded video and determine if any replacement content fortheir location is applicable while delivering the video to users. If so,then the edge server may decode portions of the encoded video, insertthe replacement content in the replacement areas, and then re-encode thevideo. The edge server then sends the encoded video with the replacementcontent to applicable client devices.

When performing the replacement in real-time while delivering the videoto client devices, the edge server decodes a portion of the video,replaces content in the video with replacement content, and thenre-encodes the portion of the video without disrupting the user'sviewing experience. The edge server may perform this process as quicklyas possible to avoid experiencing any disruptions in the video delivery.Accordingly, some embodiments provide an encoding and decoding processthat can re-use motion prediction information, which may include amacroblock type and/or motion vector. For example, a frame of video mayinclude a large amount of macroblocks, which the transcoder candetermine as different macroblock types, such as intra-frame,inter-frame and bi-direction inter-frame macroblock types. Also, thesemacroblocks also include motion vectors that are used in the predictionprocess. Given that the replacement content will replace some of themacroblocks in a frame, but not all the macroblocks, the transcoder maydetermine during the decoding process the macroblock type that was usedand may also determine the motion vector for macroblocks that are notaffected by the replacement. Then, when re-encoding the frame, themacroblock type and motion vector can be re-used. However, formacroblocks affected by the content replacement, the transcoder canre-compute the macroblock type and motion vector. However, re-using someof the macroblock type and motion vector information saves a largeamount of computation time for the transcoder.

FIG. 1 depicts a simplified system 100 for delivering video withreplacement content according to some embodiments. System 100 includes avideo delivery system 102 and client devices 104. Additionally, system100 may include a content delivery network (CDN) that may include anorigin server 112 and edge servers 114. A replacement content platform110 may provide the replacement content to the content delivery network,but other systems may deliver the replacement content to the contentdelivery network.

Video delivery system 102 may deliver video to client devices 104. Videodelivery system 102 may use different protocols to encode the video anda person skilled in the art will appreciate different protocols may usedifferent terms, such as a frame may be referred to as a picture orimage, and a macroblock may be a block in the frames.

Client devices 104 may include various devices that can view videos,such as mobile devices, set top boxes, game consoles, streaming devices,etc. Client devices 104-1 to 104-4 may be located in different locationsand may be used by different users. For example, client devices 104-1 to104-4 may be located in locations #1, #2, #3, and #4, respectively. Thedifferent locations may be defined by various characteristics, such asby location information that is defined by the Internet protocol (IP)address of client devices 104, global positioning satellite (GPS)information, or user profile information (e.g., a user's home address).

A geolocation server 106 receives the client device locations and mayclassify the locations into geolocation groups, which may be defined asa ZIP code or set of ZIP codes. In some embodiments, geolocation server106 may generate the geolocation groups dynamically based on aclustering of the locations received from active users. For example,geolocation server 106 receives the location information from activeusers that requested (and are viewing or will be viewing) a video andcan then cluster the users into different groups using a clusteringalgorithm. This may create different geolocation groups with a certainnumber of users. The geolocation group may include users that areconsidered geographically near each other, such as by a threshold orwithin a set of zip codes. The generation of real-time geolocationgroups allows targeting to users that are currently within a location.For example, ads can be delivered to users that are within ten miles ofa restaurant. Additionally, the geolocation groups may be static, suchas static groups of a set of ZIP codes, and the users are classifiedinto the different geolocation groups when their respective geolocationinformation matches the characteristics of one of the groups, such as auser's location matches a zip code of a group. Geolocation server 106may dynamically determine the geolocation groups because the contentreplacement is a computationally expensive operation and if there are nousers in the geolocation group or a sufficient number of users thatrequested the video is above a threshold, then the content replacementmay not be performed.

A content analysis server 108 may analyze the videos being offered bythe video delivery service to determine replacement content areas thatdefine where replacement content can be inserted in the video. Forexample, the areas may define insertion points and replacement contentareas in the images. The replacement content insertion points may be atime period, such as a time period in the video (from 1:01 to 1:02) orthe frame number, such as frames #100, #101, and #102. Content analysisserver 108 may use different detection tools to detect the points toinsert the content, such as using face detection and object detection todetect faces or objects that can be replaced. Content analysis server108 may also dynamically determine the insertion points in the videos.For example, depending on the geolocation groups that are active,content analysis server 108 may want to determine different content toreplace. In some examples, an advertiser may want to replace contentonly when there are a large number of users in a certain location andmay specify requirements for the replacement content areas. Thereplacement content for this advertiser may include certain dimensions,and content analysis server 108 may analyze the video to determine acontent replacement area that satisfies the replacement contentspecifications.

Also, content analysis server 108 may analyze the video before the videois sent to the users and can determine different content replacementareas that are possible. Then, when the videos are being played,different replacement content may be inserted in the content replacementareas.

Replacement content platform 110 receives the replacement content areasand geolocation groups from video delivery system 102. Replacementcontent platform 110 can then determine which content replacement areasshould be replaced with which replacement content. Replacement contentplatform 110 may make this determination based on different factors. Forexample, replacement content may only be inserted into a video if acertain number of users are within a geolocation group that are viewingthe video. Or, the ad is inserted only for users that are currentlywithin a location, such as near a restaurant or other business.

An origin server 112 of the CDN may receive the videos to deliver toclient devices 104. In some embodiments, origin server 112 may encodethe video without the replacement content. Alternatively, the videoreceived at origin server 112 may already be encoded. Then, originserver 112 sends the encoded content to edge servers 114-1 and 114-2,which may be in different locations. Edge servers 114 are located on theedge of the CDN and used to deliver the content to client devices 104.For example, edge server 114-1 may be close to location #1 and location#2 and edge server 114-2 may be close to location #3 and location #4.That is, edge server 114-2 is closer to locations #3 and #4 thanlocations #1 and location #2 and vice versa for edge server 114-1.Although origin server 112 and edge servers 114 are described, differentstructures for content delivery networks will be appreciated, such asorigin servers may not be used.

Origin server 112 may provide edge servers 114-1 and 114-2 with anindication if content should be replaced in the videos. In anotherembodiment, replacement content platform 110 may send the indicationsdirectly to edge servers 114-1 and 114-2.

When no content replacement is performed, edge servers 114-1 and 114-2may forward the encoded video to client devices 104-1 and 104-2 and104-3 and 104-4, respectively, without decoding the video. However, whenreplacement content is being inserted into the video, edge servers 114include transcoders 116 that can decode portions of the video, insertthe replacement content, and then re-encode the portions of the video.In some embodiments, edge servers 114-1 and 114-2 perform the decodingand encoding of the portions of video in real-time as the encoded videois received. For example, origin server 112 may provide segments of thevideo to edge servers 114-1 and 114-2, which then provide the segmentsto client devices 104. When receiving the segments, edge servers 114-1and 114-2 decode the portions of the video in which replacement contentwill be inserted, insert the replacement content, and then re-encode thevideo. Then, edge servers 114-1 and 114-2 send the videos with thereplacement content to respective client devices 104. For example, edgeserver 114-1 may send the video with first replacement content to clientdevices 104-1 and 104-2 and edge server 114-2 may send the video withsecond replacement content to client devices 104-3 and 104-4.

Transcoders 116-1 and 116-2 to perform the decoding, replacement, andencoding efficiently to avoid disruption to the user's viewingexperience, such as jitter or buffering of the video should be avoided.As will be described in more detail below, transcoders 116-1 and 116-2may not re-encode all frames in the video, but only the portion offrames that are affected by the replacement content. Further,transcoders 116-1 and 116-2 may re-use motion prediction information,such as macroblock type and motion vectors when re-encoding certainmacroblocks that are not affected by the replacement content. Themacroblocks affected by the replacement content may have theirmacroblock type and motion vectors recalculated. However, thereplacement content may be in a limited region of the frame and thus notrecalculating the motion prediction information for a large number ofmacroblocks in the frame may save a large amount of re-encoding time,such as two-thirds of the time.

Video Replacement Examples

FIGS. 2A and 2B depict different examples of the video that are shown indifferent locations according to some embodiments. FIG. 2A depicts anexample of client device 104-1 at location #1 according to someembodiments. Client device 104-1 includes a media player 202-1 that isplaying video content 204. Video content 204 may be a movie, show,sporting event, or other content. Also, video content 204 may includereplacement content 206-1 for location #1. For example, the replacementcontent may replace the original content in video content 204, such ascontent of a billboard may be replaced with an advertisement. Theadvertisement may be directed to users in location #1, such as thecontent may be for a local restaurant.

FIG. 2B depicts an example of client device 104-3 that is in location #3according to some embodiments. Client device 104-3 includes media player202-3 that is also playing video content 204. However, video content 204includes replacement content for location #3. The replacement contentmay be an advertisement directed to location #3, such as anotherrestaurant that is near location #3. Accordingly, edge servers 114 havereplaced content in the video with different replacement content.

Content Replacement

FIG. 3 depicts a simplified flowchart 300 of a method for replacingcontent in videos according to some embodiments. At 302, contentanalysis server 108 analyzes content for a channel. For example, videodelivery system 102 may offer different content on different channels.Content analysis server 108 may analyze the content for videos offeredon one or more of the channels. As discussed above, content analysisserver 108 may analyze content to detect content replacement areas.Additionally, content analysis server 108 may receive specifications ofcontent replacement areas, such as from the video delivery service orfrom advertisers.

At 304, content analysis server 108 stores the replacement contentareas. In some embodiments, the replacement content areas may be definedby a time period and areas in images of the video. The time period asdiscussed above may include frame numbers and the areas in the images ofthe video may include macroblock identifiers in the frames.

At 306, geolocation server 106 receives geolocation information forusers of client devices 104. Then, at 308, geolocation server 106classifies the users into different geolocation groups. The geolocationinformation may be based on which videos are being offered on channelsusers are watching. Then, a breakdown of which users in which locationsare watching which channels can be determined. Geolocation server 106can then pass the geolocation groups to content analysis server 108.

At 310, content analysis server 108 matches replacement content for thedifferent geolocation groups with the replacement content areas. Contentanalysis server 108 may select the replacement content areas based uponreplacement content that could be inserted into the replacement contentareas for the different geolocation groups. For example, if a certaingeolocation group is represented, then replacement content for thatgeolocation group may be selected, such as an ad that is local to thegeolocation group. Also, the characteristics of the replacement contentmay be taken into account in determining the replacement content areas.For example, if the geolocation group associated with a specificreplacement content that fits within a certain size replacement contentarea, then content analysis server 108 selects replacement content areasin the video that meet the requirements of the replacement content. Forexample, content analysis server 108 may select a 32×32 replacement areaif the replacement content size is 32×32 pixels. At 312, contentanalysis server 108 sends information for the replacement content andthe replacement content areas for the video and the geolocation groupsto replacement content platform 110.

Video Delivery Process

Origin server 112 receives the videos, such as from video deliverysystem 102, and can distribute the videos to client devices 104. In someembodiments, origin server 112 may encode the videos using an encodingprotocol. For example, origin server 112 may segment the videos intosegments, such as 2-10 second segments that can be requested by clientdevices 104. Although segment-based streaming protocols are discussed,other streaming protocols may be used that do not use segments may beused.

Origin server 112 can then distribute the encoded videos to edge servers114-1 and 114-2. Although the use of origin server 112 and edge servers114-1 and 114-2 is described, it will be understood that origin server112 may perform the functions as described with respect to edge servers114-1 and 114-2 or an origin server may not be used. When edge servers114-1 and 114-2 receive the encoded video from origin server 112, edgeservers 114-1 and 114-2 determine whether or not replacement contentshould be inserted into the encoded video.

FIG. 4 depicts a simplified flowchart 400 of a method for insertingreplacement content into a video according to some embodiments. Theprocess will be described with respect to edge server 114, but it willbe understood that edge server 114-1 and/or 114-2 can perform theprocess. At 402, edge server 114 receives the video, replacementcontent, one or more replacement content areas, and geolocationinformation for the replacement content. In some embodiments, all edgeservers 114 may receive the same replacement content, replacementcontent areas, and geolocation information. Then, edge servers 114decide whether or not they are in the location in which specificreplacement content should be inserted into the video. For example,geolocation information may identify client devices in location #1should receive the replacement content. Also, the geolocationinformation may state that location #1 and location #2 should receivethe replacement content. If location #1 is specified for the geolocationinformation, then when edge server 114-2 receives the replacementcontent, edge server 114-2 does not insert the replacement contentbecause client device 104-3 is in location #3 and client device 104-4 isin location #4 which is not within location #1. However, edge server114-1 determines that client 104-1 is within location #1. In otherexamples, edge servers 114 are defined by a location area, and if thegeolocation information is within the location area, then edge server114 may insert the replacement content in the video.

Accordingly, edge server 114 determines whether it is associated withclient devices 104 that are within the geolocation area specified by thegeolocation information. If the geolocation area applies to edge server114, then at 406, transcoder 116 decodes a portion of the videocorresponding to a time period for the replacement content area. Forexample, the replacement content area may specify the frame numbers or atime within the video in which the replacement content should beinserted. Transcoder 116 may determine that portion of the encoded videoand decode only that portion of the video while leaving the otherportions of the video in the encoded form.

At 408, transcoder 116 may embed the replacement content in the decodedvideo. For example, as will be described in more detail below,transcoder 116 may replace macroblocks in frames of the video with thereplacement content.

At 410, transcoder 116 re-encodes the video with the replacementcontent. For example, transcoder 116 re-encodes the frames in the timeperiod that include the replacement content. As will be discussed below,transcoder 116 may reuse motion prediction information for macroblocksthat are not affected by the replacement content insertion. At 412, edgeserver 114 splices the re-encoded content with the content that was notre-encoded and delivers the video to client devices 104 in thegeolocation area. For example, edge server 114 sends the video with thereplacement content to client devices 104 that are identified in thegeolocation group. In other embodiments, any client devices 104 withinthe geolocation group that requested the video receive the video withthe replacement content.

Transcoder

FIG. 5 depicts a more detailed example of transcoder 116 according tosome embodiments. Although this example of transcoder 116 is described,other configurations may be appreciated. Further, although selectivedecoding of portions of the video is described, it will be understoodthat the entire video may be decoded and then re-encoded with theinserted replacement content.

A content selector 502 receives the video and replacement areas. Contentselector 502 can then select the frames of the video that correspond tothe replacement areas. For example, frames in which replacement contentshould be inserted are forwarded to decoder 504. Also, content selector502 forwards frames that will not have replacement content inserted to asplicer 510.

Decoder 504 then decodes the portions of the video. As will be discussedin more detail below, decoder 504 may save some information from thedecoding process, such as motion prediction information, that can beused by encoder 508. Other information that can be used by encoder 508may also be saved, such as macroblock identifiers. This process will bedescribed in more detail below.

Content insertion engine 506 receives the decoded video and thereplacement content. Then, content insertion engine 506 inserts thereplacement content in the decoded video. For example, the replacementcontent may only replace a small area, such as a small number ofmacroblocks in each frame. The other macroblocks in the frames remainthe same as found in the original encoded video and are not replacedwith any content.

After content insertion engine 506 performs the content replacement,encoder 508 then encodes the frames of video. In some embodiments,encoder 508 may re-use motion prediction information that was receivedfrom decoder 504 and encode the frames of video. For example,macroblocks that are not affected by the replacement content may re-usemotion prediction information. This saves time in re-encoding the videoas determining the motion prediction information may be aresource-intensive operation. Once the frames are encoded, encoder 508sends the encoded video with the inserted replacement content to splicer510. Splicer 510 then splices the encoded video with the replacementcontent with the encoded video that did not have any replacement contentin it to create a video stream with portions that include thereplacement content.

Transcoder 116 may be implemented in different configurations. Forexample, depending on the protocol used, transcoder 116 may includedifferent blocks that perform different functions. FIG. 6 depicts anexample of transcoder 116 according to some embodiments. It will berecognized that some functions described may be different based upon theencoding protocol that is used.

Entropy decoder 602 receives the video and performs entropy decoding.Different entropy decoding methods may be used, such as context-adaptervariable length coding (CAVLC) or context-adapted binary arithmeticcoding (CABAC). A re-order block 604 may re-order the frames that aredecompressed in a different order than their display order.

An inverse quantization (IQ) block 606 de-quantizes the quantizedtransform coefficients of the macroblocks. An inverse transformation(IT) block 606 receives the de-quantized transform coefficients, whichare then inverse transformed.

Motion prediction block 610 then performs motion prediction. Motionprediction may include searching for a macroblock type and alsocalculating a motion vector. The macroblock type may be the type ofmacroblock that is used in the prediction, such as intra-frame,inter-frame and bi-direction inter-frame macroblock types. Inter-pictureprediction is prediction between frames and intra-picture predictionuses macroblocks only in the same frame. The motion vectors for amacroblock point to another macroblock that is used to perform theprediction. For inter-picture prediction, the motion vector points toanother macroblock in another frame and the intra-picture prediction,the motion vector points to another macroblock in the same frame.

Motion prediction block 610 then saves the motion predictioninformation, such as the macroblock type and motion vector informationat 611. Motion prediction information 611 is stored in storage for thedecoding process and later accessed by the encoding process. Filter 612can then perform filtering that prevents blocking artifacts to provide abetter visual appearance and compression efficiency. This completes thedecoding process.

The encoding process of transcoder 116 is then performed. For example,motion prediction block 614 performs motion prediction for themacroblocks. Motion prediction block 614 can re-use the motionprediction information 611 from the decoding process. For themacroblocks that are not affected by the replacement content, motionprediction block 614 may re-use the macroblock type and/or motionvector. However, for the macroblocks that are affected by thereplacement content, motion prediction block 614 performs the motionprediction process.

FIG. 7 depicts an example of performing the motion prediction processaccording to some embodiments. A first frame 700-1 is a frame that doesnot include the replacement content. A macroblock 706 is to be replacedin frame 700-1. As shown, replacement content 702 is inserted into aframe 700-2 for macroblock 706. Motion prediction information 611 may beused for macroblocks that are not affected by the replacement contentbeing inserted in frame 700-2. However, macroblocks that are affected bythe replacement content insertion have their motion predictioninformation recalculated for the encoding process.

Macroblock 706 that includes the replacement content in frame 700-2 hasits motion prediction information recalculated. However, this may not bethe only macroblock that is affected by the replacement content. Forexample, macroblocks that are close to macroblock 706 may be affected bythe replacement content. This is because the motion predictioninformation for macroblocks around macroblock 706 cannot be used if themotion prediction information is affected by the changed content inmacroblock 706. One reason the motion prediction information may not beused is a macroblock type may change or the motion vector may change dueto the inserted replacement content. Thus, the motion predictioninformation for macroblocks 708 that are around macroblock 706 may beaffected and motion prediction block 614 may calculate which macroblocksare affected by the replacement content. Accordingly, motion predictionblock 614 recalculates the motion prediction information for macroblocks708 and also for macroblock 706.

When macroblocks are not affected by the change, motion prediction block614 uses motion prediction information 611, such as motion predictioninformation for macroblocks 710 and 712, when re-encoding frame 700-2.Additionally, macroblocks other than macroblocks 706 and 708 may alsore-use the motion prediction information from the decoding processalthough this is not shown.

Referring back to FIG. 6, after motion prediction, a transformationblock performs a transform operation. Then, a quantization block 618quantizes the transform coefficients and converts the transformcoefficients into a finite number of possible values. A re-order block620 may re-order the frames. Then, entropy encoder block 622 entropyencodes the quantized coefficients, which results in final compressionbits that can be transmitted. The output of entropy encoder block 622 isthe encoded video with the replacement content.

Accordingly, particular embodiments provide a transcoder 116 that canre-use motion prediction information from the decoding process. Thisincreases the speed of the subsequent encoding process of transcoder 116after insertion of the replacement content. The motion prediction blockmay determine which macroblocks are affected by the insertion of thereplacement content and then re-use the motion prediction informationfor macroblocks that are not affected.

Other Examples

In other embodiments, edge server 114 may not perform the transcoding.Rather, origin server 112 may insert the replacement content andre-encode the video. The segments can then be sent to edge server 114,which can then forward the segments to client devices 104. However,performing the content replacement on edge server 114 may save bandwidthbetween origin server 112 and edge server 114. Bandwidth is savedbecause origin server 112 would have to send multiple streams of thevideo with the different replacement content. This may result inmultiple streams of the video being sent.

In other embodiments, client device 104 may perform the contentreplacement. In this case, transcoder 116 may reside on client device104 and perform the decoding and re-encoding.

Conclusion

Some embodiments use different replacement content based on thelocation. Accordingly, transcoder 116 determines which macroblocks areaffected by the replacement content dynamically. Accordingly, for thesame video, different edge servers 114 may be performing differentdecoding and encoding processes based on the different replacementcontent being inserted into the video.

When client devices 104 request a segment of the video, edge server 114can return a segment that includes the replacement content. Edge server114 may perform the replacement within the time required from when edgeserver 114 received the previous segment request to this request for thesegment with the replacement content. To perform the encoding in thattime, edge server 114 uses the optimization of not recalculating themotion prediction information when re-encoding the frames with thereplacement content.

System

Features and aspects as disclosed herein may be implemented inconjunction with a video streaming system 800 in communication withmultiple client devices via one or more communication networks as shownin FIG. 8. Aspects of the video streaming system 800 are describedmerely to provide an example of an application for enabling distributionand delivery of content prepared according to the present disclosure. Itshould be appreciated that the present technology is not limited tostreaming video applications, and may be adapted for other applicationsand delivery mechanisms.

In one embodiment, a media program provider may include a library ofmedia programs. For example, the media programs may be aggregated andprovided through a site (e.g., Website), application, or browser. A usercan access the media program provider's site or application and requestmedia programs. The user may be limited to requesting only mediaprograms offered by the media program provider.

In system 800, video data may be obtained from one or more sources forexample, from a video source 810, for use as input to a video contentserver 802. The input video data may comprise raw or edited frame-basedvideo data in any suitable digital format, for example, Moving PicturesExperts Group (MPEG)-1, MPEG-2, MPEG-4, VC-1, H.264/Advanced VideoCoding (AVC), High Efficiency Video Coding (HEVC), or other format. Inan alternative, a video may be provided in a non-digital format andconverted to digital format using a scanner and/or transcoder. The inputvideo data may comprise video clips or programs of various types, forexample, television episodes, motion pictures, and other contentproduced as primary content of interest to consumers. The video data mayalso include audio or only audio may be used.

The video streaming system 800 may include one or more computer serversor modules 802, 804, and/or 807 distributed over one or more computers.Each server 802, 804, 807 may include, or may be operatively coupled to,one or more data stores 809, for example databases, indexes, files, orother data structures. A video content server 802 may access a datastore (not shown) of various video segments. The video content server802 may serve the video segments as directed by a user interfacecontroller communicating with a client device. As used herein, a videosegment refers to a definite portion of frame-based video data, such asmay be used in a streaming video session to view a television episode,motion picture, recorded live performance, or other video content.

In some embodiments, a video advertising server 804 may access a datastore of relatively short videos (e.g., 10 second, 30 second, or 60second video advertisements) configured as advertising for a particularadvertiser or message. The advertising may be provided for an advertiserin exchange for payment of some kind, or may comprise a promotionalmessage for the system 800, a public service message, or some otherinformation. The video advertising server 804 may serve the videoadvertising segments as directed by a user interface controller (notshown).

The video streaming system 800 may further include an integration andstreaming component 807 that integrates video content and videoadvertising into a streaming video segment. For example, streamingcomponent 807 may be a content server or streaming media server. Acontroller (not shown) may determine the selection or configuration ofadvertising in the streaming video based on any suitable algorithm orprocess. The video streaming system 800 may include other modules orunits not depicted in FIG. 8, for example administrative servers,commerce servers, network infrastructure, advertising selection engines,and so forth.

The video streaming system 800 may connect to a data communicationnetwork 812. A data communication network 812 may comprise a local areanetwork (LAN), a wide area network (WAN), for example, the Internet, atelephone network, a wireless cellular telecommunications network (WCS)814, or some combination of these or similar networks.

One or more client devices 820 may be in communication with the videostreaming system 800, via the data communication network 812 and/orother network 814. Such client devices may include, for example, one ormore laptop computers 820-1, desktop computers 820-2, “smart” mobilephones 820-3, tablet devices 820-4, network-enabled televisions 820-5,or combinations thereof, via a router 818 for a LAN, via a base station817 for a wireless telephony network 814, or via some other connection.In operation, such client devices 820 may send and receive data orinstructions to the system 800, in response to user input received fromuser input devices or other input. In response, the system 800 may servevideo segments and metadata from the data store 809 responsive toselection of media programs to the client devices 820. Client devices820 may output the video content from the streaming video segment in amedia player using a display screen, projector, or other video outputdevice, and receive user input for interacting with the video content.

Distribution of audio-video data may be implemented from streamingcomponent 807 to remote client devices over computer networks,telecommunications networks, and combinations of such networks, usingvarious methods, for example streaming. In streaming, a content serverstreams audio-video data continuously to a media player componentoperating at least partly on the client device, which may play theaudio-video data concurrently with receiving the streaming data from theserver. Although streaming is discussed, other methods of delivery maybe used. The media player component may initiate play of the video dataimmediately after receiving an initial portion of the data from thecontent provider. Traditional streaming techniques use a single providerdelivering a stream of data to a set of end users. High bandwidths andprocessing power may be required to deliver a single stream to a largeaudience, and the required bandwidth of the provider may increase as thenumber of end users increases.

Streaming media can be delivered on-demand or live. Streaming enablesimmediate playback at any point within the file. End-users may skipthrough the media file to start playback or change playback to any pointin the media file. Hence, the end-user does not need to wait for thefile to progressively download. Typically, streaming media is deliveredfrom a few dedicated servers having high bandwidth capabilities via aspecialized device that accepts requests for video files, and withinformation about the format, bandwidth and structure of those files,delivers just the amount of data necessary to play the video, at therate needed to play it. Streaming media servers may also account for thetransmission bandwidth and capabilities of the media player on thedestination client. Streaming component 807 may communicate with clientdevice 820 using control messages and data messages to adjust tochanging network conditions as the video is played. These controlmessages can include commands for enabling control functions such asfast forward, fast reverse, pausing, or seeking to a particular part ofthe file at the client.

Since streaming component 807 transmits video data only as needed and atthe rate that is needed, precise control over the number of streamsserved can be maintained. The viewer will not be able to view high datarate videos over a lower data rate transmission medium. However,streaming media servers (1) provide users random access to the videofile, (2) allow monitoring of who is viewing what video programs and howlong they are watched, (3) use transmission bandwidth more efficiently,since only the amount of data required to support the viewing experienceis transmitted, and (4) the video file is not stored in the viewer'scomputer, but discarded by the media player, thus allowing more controlover the content.

Streaming component 807 may use TCP-based protocols, such as HTTP andReal Time Messaging Protocol (RTMP). Streaming component 807 can alsodeliver live webcasts and can multicast, which allows more than oneclient to tune into a single stream, thus saving bandwidth. Streamingmedia players may not rely on buffering the whole video to providerandom access to any point in the media program. Instead, this isaccomplished through the use of control messages transmitted from themedia player to the streaming media server. Another protocol used forstreaming is hypertext transfer protocol (HTTP) live streaming (HLS) orDynamic Adaptive Streaming over HTTP (DASH). The HLS or DASH protocoldelivers video over HTTP via a playlist of small segments that are madeavailable in a variety of bitrates typically from one or more contentdelivery networks (CDNs). This allows a media player to switch bothbitrates and content sources on a segment-by-segment basis. Theswitching helps compensate for network bandwidth variances and alsoinfrastructure failures that may occur during playback of the video.

The delivery of video content by streaming may be accomplished under avariety of models. In one model, the user pays for the viewing of videoprograms, for example, using a fee for access to the library of mediaprograms or a portion of restricted media programs, or using apay-per-view service. In another model widely adopted by broadcasttelevision shortly after its inception, sponsors pay for thepresentation of the media program in exchange for the right to presentadvertisements during or adjacent to the presentation of the program. Insome models, advertisements are inserted at predetermined times in avideo program, which times may be referred to as “ad slots” or “adbreaks.” With streaming video, the media player may be configured sothat the client device cannot play the video without also playingpredetermined advertisements during the designated ad slots.

Referring to FIG. 9, a diagrammatic view of an apparatus 900 for viewingvideo content and advertisements is illustrated. In selectedembodiments, the apparatus 900 may include a processor (CPU) 902operatively coupled to a processor memory 904, which holds binary-codedfunctional modules for execution by the processor 902. Such functionalmodules may include an operating system 906 for handling systemfunctions such as input/output and memory access, a browser 908 todisplay web pages, and media player 910 for playing video. The memory904 may hold additional modules not shown in FIG. 9, for example modulesfor performing other operations described elsewhere herein.

A bus 914 or other communication component may support communication ofinformation within the apparatus 900. The processor 902 may be aspecialized or dedicated microprocessor configured to perform particulartasks in accordance with the features and aspects disclosed herein byexecuting machine-readable software code defining the particular tasks.Processor memory 904 (e.g., random access memory (RAM) or other dynamicstorage device) may be connected to the bus 914 or directly to theprocessor 902, and store information and instructions to be executed bya processor 902. The memory 904 may also store temporary variables orother intermediate information during execution of such instructions.

A computer-readable medium (CRM) in a storage device 924 may beconnected to the bus 914 and store static information and instructionsfor the processor 902; for example, the storage device (CRM) 924 maystore the modules 906, 908, and 910 when the apparatus 900 is poweredoff, from which the modules may be loaded into the processor memory 904when the apparatus 900 is powered up. The storage device 924 may includea non-transitory computer-readable storage medium holding information,instructions, or some combination thereof, for example instructions thatwhen executed by the processor 902, cause the apparatus 900 to beconfigured to perform one or more operations of a method as describedherein.

A communication interface 916 may also be connected to the bus 914. Thecommunication interface 916 may provide or support two-way datacommunication between the apparatus 900 and one or more externaldevices, e.g., the streaming system 800, optionally via a router/modem926 and a wired or wireless connection. In the alternative, or inaddition, the apparatus 900 may include a transceiver 918 connected toan antenna 929, through which the apparatus 900 may communicatewirelessly with a base station for a wireless communication system orwith the router/modem 926. In the alternative, the apparatus 900 maycommunicate with a video streaming system 800 via a local area network,virtual private network, or other network. In another alternative, theapparatus 900 may be incorporated as a module or component of the system800 and communicate with other components via the bus 914 or by someother modality.

The apparatus 900 may be connected (e.g., via the bus 914 and graphicsprocessing unit 920) to a display unit 928. A display 928 may includeany suitable configuration for displaying information to an operator ofthe apparatus 900. For example, a display 928 may include or utilize aliquid crystal display (LCD), touchscreen LCD (e.g., capacitivedisplay), light emitting diode (LED) display, projector, or otherdisplay device to present information to a user of the apparatus 900 ina visual display.

One or more input devices 930 (e.g., an alphanumeric keyboard,microphone, keypad, remote controller, game controller, camera or cameraarray) may be connected to the bus 914 via a user input port 922 tocommunicate information and commands to the apparatus 900. In selectedembodiments, an input device 930 may provide or support control over thepositioning of a cursor. Such a cursor control device, also called apointing device, may be configured as a mouse, a trackball, a track pad,touch screen, cursor direction keys or other device for receiving ortracking physical movement and translating the movement into electricalsignals indicating cursor movement. The cursor control device may beincorporated into the display unit 928, for example using a touchsensitive screen. A cursor control device may communicate directioninformation and command selections to the processor 902 and controlcursor movement on the display 928. A cursor control device may have twoor more degrees of freedom, for example allowing the device to specifycursor positions in a plane or three-dimensional space.

Particular embodiments may be implemented in a non-transitorycomputer-readable storage medium for use by or in connection with theinstruction execution system, apparatus, system, or machine. Thecomputer-readable storage medium contains instructions for controlling acomputer system to perform a method described by particular embodiments.The computer system may include one or more computing devices. Theinstructions, when executed by one or more computer processors, may beconfigured to perform that which is described in particular embodiments.

As used in the description herein and throughout the claims that follow,“a”, “an”, and “the” includes plural references unless the contextclearly dictates otherwise. Also, as used in the description herein andthroughout the claims that follow, the meaning of “in” includes “in” and“on” unless the context clearly dictates otherwise.

The above description illustrates various embodiments along withexamples of how aspects of particular embodiments may be implemented.The above examples and embodiments should not be deemed to be the onlyembodiments, and are presented to illustrate the flexibility andadvantages of particular embodiments as defined by the following claims.Based on the above disclosure and the following claims, otherarrangements, embodiments, implementations and equivalents may beemployed without departing from the scope hereof as defined by theclaims.

What is claimed is:
 1. A method comprising: receiving, by a computingdevice, an encoded video; decoding, by the computing device, blocks in aframe of the encoded video in which replacement content will beinserted; analyzing, by the computing device, the blocks to determinefirst blocks of the frame that are affected by insertion of thereplacement content and second blocks in the frame that are not affectedby the insertion, wherein information from the decoding of the block issaved for the second blocks; and encoding, by the computing device, theframe, wherein the replacement content is inserted to replace the firstblocks in the encoding of the frame, and wherein the information fromthe decoding of the second blocks is reused when performing the encodingof the second blocks.
 2. The method of claim 1, further comprising:determining a location associated with a client device that requested toview the encoded video, wherein the location is used to select thereplacement content.
 3. The method of claim 2, wherein: the locationcomprises a first location, and the encoded video at a second locationhas different replacement content inserted in the encoded video toreplace the first blocks.
 4. The method of claim 2, further comprising:receiving a geolocation group for the replacement content; anddetermining that the location associated with the client device iswithin the geolocation group.
 5. The method of claim 4, wherein thegeolocation group is determined based on analyzing active users thatrequested the encoded video.
 6. The method of claim 2, furthercomprising: receiving a geolocation group for different replacementcontent; and determining that the location associated with the clientdevice is not within the geolocation group, wherein the differentreplacement content is not inserted in the encoded video to replace thefirst blocks.
 7. The method of claim 1, wherein: the information fromdecoding the blocks comprises motion prediction information for at leastsome of the second blocks, and the encoding of the second blocks reusesthe motion prediction information for the at least some of the secondblocks.
 8. The method of claim 7, further comprising: storing the motionprediction information for the second blocks.
 9. The method of claim 1,wherein: the information from decoding the blocks includes a block typefor at least some of the second blocks, and the encoding of the secondblocks reuses the block type for the at least some of the second blocks.10. The method of claim 1, wherein: the information from decoding theframe includes a motion vector for at least some of the second blocks,and the encoding of the second blocks reuses the motion vector for theat least some of the second blocks.
 11. The method of claim 1, wherein:the information from decoding the blocks comprises macroblockidentifiers for the second blocks, and the encoding of the second blocksreuses the macroblock identifiers for the at least some of the secondblocks.
 12. The method of claim 1, wherein encoding the frame comprises:inserting the replacement content in the first blocks of the frame; andencoding the first blocks and the second blocks of the frame.
 13. Themethod of claim 1, further comprising: splicing the frame with thereplacement content with other frames that do not include replacementcontent; and sending the frame and the other frames to a client device.14. The method of claim 1, wherein analyzing the blocks comprises:selecting a first portion of the first blocks based on the replacementcontent replacing the first portion of the first blocks; and selecting asecond portion of the first blocks based on the second portion of thefirst blocks having motion prediction information that is affected bythe replacing of the first portion of the first blocks.
 15. The methodof claim 1, wherein the encoded video is received from an origin serverwith information for the replacement content.
 16. The method of claim 1,further comprising: receiving a replacement content area for thereplacement content; and selecting the frame and the second blocks ofthe encoded video based on the replacement content area.
 17. Anon-transitory computer-readable storage medium containing instructions,that when executed, control a computer system to be configured for:receiving an encoded video; decoding blocks in a frame of the encodedvideo in which replacement content will be inserted; analyzing theblocks to determine first blocks of the frame that are affected byinsertion of the replacement content and second blocks in the frame thatare not affected by the insertion, wherein information from the decodingof the block is saved for the second blocks; and encoding the frame,wherein the replacement content is inserted to replace the first blocksin the encoding of the frame, and wherein the information from thedecoding of the second blocks is reused when performing the encoding ofthe second blocks.
 18. The non-transitory computer-readable storagemedium of claim 17, the information from decoding the blocks comprisesmotion prediction information for at least some of the second blocks,and the encoding of the second blocks reuses the motion predictioninformation for the at least some of the second blocks.
 19. Thenon-transitory computer-readable storage medium of claim 17, wherein:the information from decoding the frame includes a motion vector for atleast some of the second blocks, and the encoding of the second blocksreuses the motion vector for the at least some of the second blocks. 20.An apparatus comprising: one or more computer processors; and anon-transitory computer-readable storage medium comprising instructions,that when executed, control the one or more computer processors to beconfigured for: receiving an encoded video; decoding blocks in a frameof the encoded video in which replacement content will be inserted;analyzing the blocks to determine first blocks of the frame that areaffected by insertion of the replacement content and second blocks inthe frame that are not affected by the insertion, wherein informationfrom the decoding of the block is saved for the second blocks; andencoding the frame, wherein the replacement content is inserted toreplace the first blocks in the encoding of the frame, and wherein theinformation from the decoding of the second blocks is reused whenperforming the encoding of the second blocks.